Device and method for capturing, analyzing, and sending still and video images of the fundus during examination using an ophthalmoscope

ABSTRACT

The present invention is directed to a medical imaging binocular indirect ophthalmoscope with onboard sensor array and computational processing unit, enabling simultaneous or time-delayed viewing and collaborative review of photographs or videos from an eye examination. The invention also claims a method for photographing and integrating information associated with the images, videos, or other data generated from the eye examination.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of and relies on thedisclosures of and claims priority to and the benefit of the filingdates of U.S. patent application Ser. No. 16/908,235, filed Jun. 22,2020 and Ser. No. 16/459,552, filed Jul. 1, 2019, U.S. patentapplication Ser. No. 15/892,286, filed Feb. 8, 2018 (now patented asU.S. Pat. No. 10,376,142), which claims priority to and the benefit ofU.S. Provisional Application No. 62/456,630, filed Feb. 8, 2017. Thedisclosures of those applications are hereby incorporated by referenceherein in their entireties.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention is directed to a medical imaging device withonboard sensor array and computational processing unit, namely abinocular indirect ophthalmoscopes (“BIOs”), enabling enhanceddiagnostic capabilities to ophthalmologists and optometrists beyond thetraditional manual ophthalmic examination, such as wireless automaticcapture and transmission of high-fidelity images directly from theperspective of a user performing an eye examination; while allowing theunimpaired, full use of the examination instrument via a customform-fitted mechanical and optical design; and enabling simultaneous ortime-delayed viewing and collaborative review of photographs or videosfrom said eye examination. The invention also includes an integratedsystem for onboard detection and enhancement of clinical imagery withambient examination-related feedback to the user via visual andnon-visual interactive notifications to aid in the diagnosticexamination, as well as the coordinated collection, transmission,management, and maintenance of imaging and related metadata fromophthalmic examinations, and additionally allows for multi-usercollaboration generated by one or more device(s) or networks of devicesand multiple users.

The present invention is directed to a medical imaging device withonboard sensor array and computational processing unit, namely a slitlamp biomicroscope (“slit lamp” or “SL”), enabling enhanced diagnosticcapabilities to ophthalmologists and optometrists beyond the traditionalmanual ophthalmic examination, such as wireless automatic capture andtransmission of high-fidelity images directly from the perspective of auser performing an eye examination; while allowing the unimpaired, fulluse of the examination instrument via a custom form-fitted mechanicaland optical design; and enabling simultaneous or time-delayed viewingand collaborative review of photographs or videos from said eyeexamination. The invention also includes an integrated system foronboard detection and enhancement of clinical imagery and onboardadditional slit lamp-mounted diagnostic testing with ambientexamination-related feedback to the user via visual and non-visualinteractive notifications to aid in the diagnostic examination, as wellas the coordinated collection, transmission, management, and maintenanceof imaging and related metadata from ophthalmic examinations, andadditionally allows for multi-user collaboration generated by one ormore device(s) or networks of devices and multiple users, detection andtagging of imagery along with the portion of the clinical examinationconducted, with transmission of imagery and related metadata to aseparate bioinformatics database (“datahub”) to pair capturedexamination data with related imagery and diagnostic testing data fromone or more other ophthalmic diagnostic instruments for the samepatient. In an embodiment, slit lamp camera photographs and relateddiagnostic testing of the anterior segment of the eye can be capturedusing the described slit lamp-mounted device and system, and wirelesslyfiled, paired, and registered with corresponding posterior segmentimagery and related diagnostic testing captured using an indirectophthalmoscope-mounted or -integrated camera or diagnostic testingdevice for the same eye of the same patient, permitting front-to-backautomated diagnostic testing and fundus photography of each patient fromthe clinical examination lane.

Further, the present invention is directed to a system and method ofusing generalizable machine learning and artificial intelligencealgorithms as an ophthalmic disease detection system using anophthalmoscope- or biomicroscope-based imaging system and machinelearning-based tools to automatically perform quality control, imagesegmentation, and estimation of eye disease risk. This system may use,in an embodiment, an integrated or adapter digital imaging device tocomprise a binocular indirect ophthalmoscope-based or -mounted system,or in another embodiment, an integrated or adapter digital imagingdevice to comprise a slit lamp biomicroscope-based or -mounted systemwith an associated on-device or off-device computing device-deployedsoftware application for ophthalmic disease screening, detection, anddiagnosis. In an embodiment, the system can be used to control forimaging artifacts currently encountered by the examiner conducting aconventional eye examination using a binocular indirect ophthalmoscopeor slit lamp biomicroscope. These imaging artifacts currently observedin captured imagery may include (but are not limited to) Purkinje lightreflections from the cornea and tear film, digital noise, light bloomand glare, poor focus, off-centered or poorly-registered ocularstructures and pathology, motion blur, shadowing or off-centeredillumination, the examiner's hands or non-ocular structures captured bythe imaging system camera. Additionally, in an embodiment, the systemwould use algorithmic- and/or sensor-based methods to detect anddetermine the distance from the instrument to the patient and ocularstructures examined, the refractive state of the eye, and the nature andpower of examination lenses used (such as, but not limited to, 20diopter, 28 diopter, 90 diopter, 60 diopter, 66 diopter, and 78 diopterhandheld condensing lenses). In another embodiment, the detection of thehandheld lens used would facilitate determination of the size ofophthalmic structures captured in the digital imagery, for recording andsubsequent analysis such as in associated clinical decision supporttools and algorithms. The system could also use robust machine learning(“ML”) algorithms to classify or detect various ophthalmic structures oreye diseases, including but not limited to glaucoma, diabeticretinopathy, retinopathy of prematurity, and age related maculardegeneration. In an embodiment, the system could screen for or detectglaucoma from retinal fundus images with high to low image quality,based on estimated Disc Damage Likelihood Score (DDLS) values fromautomatically segmented optic cup and optic disc measurements extractedfrom the fundus images. Additionally, in an embodiment, the systemaddresses the challenges of data bias/shifts caused by different imagingdevices or populations of patients; hence enforcing a fair performanceof model prediction among patients from diverse regional, racial, orethnic backgrounds. The robust nature of this system would enable higherperformance of algorithmic eye disease detection from a variety ofdigital camera, instrument, and device types, and would be comparativelyresistant to the skill level of the examiner capturing the ophthalmicimagery, which may vary widely in real-world clinical practice.

Description of the Related Art

The state of the art prior to the invention taught herein was for manualbinocular indirect ophthalmoscopes and handheld lenses to view the humaneye by a practitioner performing an eye examination, specifically alarge portion of the fundus (i.e., the retina, vitreous, and opticnerve). Given the optical properties of indirect ophthalmoscopy, theexaminer is required to mentally invert the observed retinal imageupside down and backwards to determine the correct orientation andlocation of observed findings, with rapidly-shifting focus, significantglare, subtle tilting of the handheld lens, and small shifts in patientgaze and lighting conditions often resulting in a fleeting, limited, anddistorted views of the fundus and repeated, lengthy, and uncomfortableexaminations for the patients in order to verify possible clinicalobservations through pharmacologically dilated pupils with intenselybright light from the ophthalmoscope. These prior art instruments,moreover, were purely optical, manual instruments, and unable to capturephotographs or videos in real-time during the patient examination.Recording clinical observations into the patient medical record withsuch prior art instruments requires the examiner to manually draw anartistic representation of findings as they recall their appearance(while inverting and reversing the optical appearance of findingsthrough the instrument), or describing such observations in text form.This leads to poor quality of care, due to the inherent limitations ofthe traditional fundus exam as discussed above, which require technicalstaff availability and separate imaging devices typically locatedoutside the doctor's examination room (which may or may not be readilyavailable) in order to conduct clinical photography of the eye.

More recently, a wired video indirect ophthalmoscope has been describedand produced, allowing for digital transmission of photographsconcurrent with the patient examination. However, in practice, theseinstruments typically require two people to conduct such operation,including manually focusing and aiming the onboard camera to betteralign the imaged area with the doctor's view; and managing a wired USBconnection to a computer device as well as a separate wired connectionto a footswitch controller to trigger image or video capture (thustethering the physician to both computer and footswitch, and posing ausability challenge and potential safety hazard in a dynamic clinicalexamination which requires significant movement and repositioning of theexaminer fully around on both sides of the patient to examine both eyesin all gazes of view); along with requiring the simultaneous operationof image management or video editing software with the examiner andassistant having to refer dynamically to video output on a separatescreen while the field of view of the examiner is largely taken up bythe ophthalmoscope's viewing eyepieces, all greatly diminishing theusability and practicability in practice during the patient examination,particularly in the common circumstances in which such trainedassistants are not available during the fundus examination. The inherentusability concerns and additional personnel requirements of such systemshave greatly hindered such video ophthalmoscopes' usage and spread ineye care and limited the public health benefits of concurrent digitaldocumentation of eye examinations. Additionally, the optical arrangementof key components in such prior devices use two or more triangularreflecting mirror blocks enclosed in the instrument straddling adjacentto the central visual axis of the instrument's optical system, incoordination with a centrally-positioned beamsplitter(partially-reflective mirror) in between the two triangular mirrorblocks. The centrally-positioned beamsplitter or prism (such as apentaprism) reflects the central imagery towards a centrally-positionedcamera, but the two adjacent triangular mirror blocks reflect adjacentimagery towards each respective eyepiece. As such, the optical systembears significant complexity of design, and importantly, thecentrally-positioned beamsplitter or prism may partially or totallyocclude portions of the imagery relative to the laterally-locatedoptical pathway to the instrument eyepieces, and cannot in allcircumstances ensure good correspondence between camera and examinerviews. As a result, this requires additional mechanical elements to moreclosely align the camera view with the examiner's view. However, theexaminer's view is nearly totally occluded by the instrument eyepieces.Additionally, in many instances, the eyepieces include low-plusspherical lenses (such as +1.50 or +2.00 diopters) to allow a presbyopicexaminer to comfortably view the aerial image of the examined retinathrough the handheld lens used in indirect ophthalmoscopy. In thesesituations, switching one's view through the instrument eyepieces to anoff-instrument display screen rapidly to guide the assistant orindependently attempt to adjust instrument controls to improvedevice-instrument correspondence, adjust video or image recording orplayback using onscreen controls, or interact with software userinterface elements on an external or even adjacent display may beuncomfortable or disorienting for the examiner. As such, the examiner ofsuch instruments requires an assistant to manually adjust the cameraview using such additional mechanical adjustments such as dials orlevers using an offscreen display, close communication with theexaminer, and a process of trial and error to minimize the discrepanciesbetween the examiner and camera views. Additionally, the great expenseof previous video ophthalmoscopes (on a par with benchtop funduscameras) typically only allows for installation in one examination lanefor a typical ophthalmic practice, not only providing inferior qualityimages than a benchtop fundus camera due to inherent limitations inindirect ophthalmoscopic image quality as well as previously-discussedlimitations, but also requiring the physician to capture all images(technicians are typically not trained in indirect ophthalmoscopy) andrepresenting a bottleneck in the clinical workflow of a modernophthalmology or optometry practice, as all patients in a typicalmulti-lane setting to be imaged would have to be redirected to aspecific examination lane following a traditional examination.

The remaining alternatives other than an ophthalmoscope with an embeddedcamera in order to obtain clinical photography are in using traditionalbenchtop fundus cameras and scanning laser ophthalmoscopes, both ofwhich are expensive, bulky, and require the use of a trained retinalphotographer. Nonmydriatic fundus cameras and scanning laserophthalmoscopes, which do not require the pharmacologic dilation of thepatient's pupil, can incentivize physicians to not dilate theirpatients, and thus not examine the retinal far periphery; however,nonmydriatic fundus photographs routinely produce image artifacts whichcan be read as false positives for alarming features such as retinaldetachments and posterior segment tumors, while being unable to examinethe retinal far periphery, disincentivizing eye physicians fromconducting a complete dilated examination of the ophthalmic fundus andleading to inferior quality care. Additionally, communication gapsbetween eye physicians and retinal photographers routinely limit thequality and accuracy of images actually obtained in clinical practice,and staffing and patient scheduling gaps limit whether photography canbe conducted at all using such technician-performed techniques.Smartphone and other mobile camera systems are alternatives to the fixedfundus camera, but face relatively poor patient acceptance, are oftencumbersome to use (particularly in focus and exposure control), requireyet another proprietary device to capture photographs redundant to theretinal examination, and typically also cannot adequately capture theretinal periphery. The present invention promotes the gold standardtechnique for retinal examination and the only technique allowing fordynamic examination of the full retinal periphery (indirectophthalmoscopy), as recommended by both optometric and ophthalmologyprofessional bodies, and the added benefit of simultaneous capture andredisplay of fundus photography and video, using the same existingexamination instruments doctors already possess, are trained upon, andare comfortable using, allowing for augmented retinal examinationwithout introducing a separate step in clinical image acquisition beyondthe doctor's own examination process.

Moreover, neither of the aforementioned options—traditional benchtopfundus cameras and scanning laser ophthalmoscopes—are capable of imagingthe full periphery of the retina where many retinal detachments andseveral kinds of posterior segment tumors originate—this is somethingonly indirect ophthalmoscopes are currently capable of fully viewing(e.g., the retinal far periphery), in combination with dynamic fundusexamination procedures familiar to ophthalmologists and optometrists,such as scleral depressed examination techniques, which can be performedin concert with other diagnostic and therapeutic maneuvers such asfluorescein angiography, scleral indentation, and indirect laserphotocoagulation.

In regards to SL systems, existing SL cameras are wired and mayinterfere with operation of the SL; often they have onboard screens thatmay be distracting, produce significant glare to the examiner in adarkened exam room, and obstruct the view of the examiner through the SLoculars; ocular-mounted designs do not correspond to the examiner'sbinocular view, given they typically adapt to only one of the oculars(ocular-based beamsplitter design or obstructing design such as asmartphone camera adapter occluding one or both oculars); focus andexposure captured in still photographs and/or video by the camera systemmay not correspond to the depth of focus or dynamic range appreciated bythe examining physician or technician.

Operation of existing SL cameras may be awkward, requiring a user toshift attention away from the examination.

Filing, pairing, tagging, and registration of images and metadatabetween patients, eyes, additional or corresponding diagnostic tests,and examinations is awkward, difficult, or not feasible.

At present, beyond the slit lamp clinical examination of the patient(“slit lamp exam”), additional diagnostic testing instruments andrelated imaging, patient rooming, data export, filing, organization, andrelated processes all require separate slit lamp-mounted instruments andworkflows, technician-driven processes, or locations. Existing optionsoften require switching tasks and workflows between one or morediagnostic imaging and testing devices, and requires the user tointeract with one or more separate computer interface systems with akeyboard and mouse, such as a desktop, laptop, or tablet computerrunning an electronic health record (EHR) system and/or picturearchiving and communication system (“image PACS system”) formanipulating slit lamp-based (or non-slit lamp based) ophthalmicphotography equipment and/or ophthalmic diagnostic testing equipment andrelated settings, and data entry and modification; clinicaldocumentation entry, editing, and digital authentication; navigation,retrieval, and manipulation of on-device or off-device bioinformaticsdatabases, diagnostic testing data, and additional digital examinationimagery.

In regards to AI/ML aspects of the invention as described herein,advanced techniques such as tabletop stereoscopic red-free, discphotographs, automated visual field testing, and OCT of the optic nerveretinal nerve fiber layer permit timely eye disease screening anddetection, including glaucoma screening. However, the vast majority ofscreening is done in commercial and retail optometry offices, many ofwhich have room, staff, and budget shortages for advanced diagnosticequipment. Along with the hardware shortage, existing artificialintelligence (“AI”) models to detect eye diseases such as glaucoma aremostly designed for advanced equipment, such less suited for the generalpopulation as well as optometry/ophthalmology users, not interpretable(do not permit the clinician to critically assess model performance atthe test level), and training models that may not be suitable for thedarker fundus of patients with darker skin (generalizability problems inmore diverse patient populations).

SUMMARY OF THE INVENTION

A core problem solved by the current invention is in gaining the abilityto seamlessly take clinical photographs with a high degree of fidelityfrom the ophthalmic fundus examination—in a preferred embodiment, of thepharmacologically-dilated eye of a patient—taken by the user withminimal or no modifications or alterations necessary in the examiner'sroutine examination technique. The current invention also encourages thegold-standard examination technique of the retina—indirectophthalmoscopy—and makes possible seamless wireless transmission ofclinical photographs and videos from the clinical examination to otherviewers such as students or other practitioners, for training, patientcare, clinical research, telemedicine, and feedback during or after anexamination, via integrated onboard and off-device networkedcomputational and software elements. This transforms the traditionalmanual fundus examination into an augmented image-guided examination,allowing the doctor and patient the benefits of clinical photographicdocumentation and enhancement, while optimizing workflows.

Improvements over the prior art include, but are not limited to, theability for the user to simultaneously capture ophthalmic featuresmanually; automatic device capture of images integrating onboardintegrated sensors, computational processing capabilities, andtightly-integrated on-device and off-device algorithmic processing ofimagery; allowing for feature recognition of the eye and ocular featuresof interest; and/or automatically montaging multiple overlapping imagesto more broadly map and redisplay the ocular features via networkedsoftware programs; all included in an ophthalmoscope without specializedtraining or tools. In aspects, the device taught herein improves uponprior optical design by simplifying the optical system by requiring onlyone centrally-positioned triangular reflecting mirror block (as istypically found inside the optical system of most BIOS) and an onboardlinear plate beamsplitter, not requiring a prism or laterally-locatedmirror blocks; as opposed to two or more laterally-positioned mirrorblocks in coordination with a prism (such as a pentaprism) orcentrally-positioned or laterally-positioned beamsplitter, whichordinarily may totally or partially occlude the optical pathway to theinstrument eyepieces. Such prior configurations introduce significantdevice complexity to the mechanical design and opportunities fordistortion of imagery, and cannot ensure consistent correspondencebetween camera and examiner views in many examination scenarios. Theoptical system here described allows for a greater fidelity ofcorrespondence between onboard camera and examiner views over amuch-greater breadth of examination scenarios. In one embodiment, amechanical adjustment lever allows for customization of the opticalsystem by tilting, in a coplanar relationship, the beamsplitter mirrorand embedded camera assembly, to permit a greater range of examinerwearing patterns of the instrument, such as physicians who wear theinstrument at a significant downward tilt to examine patientssignificantly below their eye level, or physicians who wear spectacles.In another embodiment, the use of a wider field of view camera lens,along with a high-resolution camera sensor, may be used to customize theviewing angle of the captured imagery to the preferred instrumentviewing position of individual users, by setting the desired viewingregion by cropping out extraneous imagery via a software interface. Inone instance, this software-enabled field of view image crop control maybe used in combination with an initial user calibration procedure using,but not requiring, the use of a standardized target to allow forautomatic configuration of the camera view—without requiring interactionwith the onboard mechanical device controls such as adjustment levers ordials.

Beyond its optical properties, the device and integrated systemdescribed also allow for subsequent review and enhancement of ocularimagery beyond the range of view ordinarily visible or accessible to theotherwise unaided eye of the examiner using their existing manual BIOinstrument. The apparatus thus permits the routine use of image-guidedaugmented examination for improved patient care, enhanced detection anddocumentation of clinical pathology, and serial comparison of follow-upfindings, as well as improved practice efficiencies by the simultaneousdual use of BIO instruments for both examination and enhanced clinicalimaging purposes.

The design allows for transmission of data, an image trigger thatautomatically captures when certain features are present in theophthalmoscope's viewfinder, manual focus, closed-loop and open-loopautofocus, and other optical and sensor-assisted algorithmic techniquessuch as focus stacking, software-selectable focus planes, expanded depthof field imaging, and region of interest (ROI)-based focus and exposurecontrols to ensure crisp focus and exposure without or with minimal userintervention in routine clinical examination settings, imageenhancement, automatic montaging to see more complete pictures of, forexample, the retina, and annotation of findings. Focus stacking issimilar to expanded depth of field imaging; also known as focal planemerging and z-stacking or focus blending; it is a digital imageprocessing technique which combines multiple images taken at differentfocus distances to give a resulting image with a greater depth of field(DOF) than any of the individual source images. Focus stacking can beused in any situation where individual images have a very shallow depthof field, such as encountered in indirect ophthalmoscopy. Unlikeexisting prior art, in one embodiment, no display screen (eitheron-device or off-device) is required to be referenced during examinationsessions for clinical findings to be imaged with high fidelity, asfocus, exposure, and field of view are all faithfully maintained by thedesign of the device. Though one prior art system describes review ofimages on the device itself on a display screen either attached orlocated adjacent to the viewpiece or handheld, given that the indirectophthalmoscope viewpieces themselves inherently occlude and take up mostof the examiner's field of view, referring to any external displayscreens while actively examining a patient is quite difficult inpractice. For this reason, no external or on-device display screen isrequired in this device design, and device notifications are designed tooccur “ambiently,” that is, without obstructing or minimally obstructingthe examiner's view, not requiring the examiner to alter their gazeduring the clinical examination and still receive examination- anddevice status-related interactive notifications. Ambient notificationsinclude, but are not limited to, light feedback, sound feedback, hapticfeedback, touch feedback, vibrations, buzzes, clicks, noises, beeps, orspeech, or any other notification or feedback that does not interferewith the examiner's examination of the patient's eye, such as byoccluding or obstructing the examiner's view or otherwise distractingthe examiner or limiting the examiner's movement.

Software-selectable focusing planes are an algorithmic and optical setof techniques in which one or more focal planes can be selected from oneor more electronically-controlled cameras, lenses, or image sensors, inwhich, in a preferred embodiment, an electronically-adjustable focallength can be selected or tuned electronically for anelectronically-tunable lens, or desired focal length can be selectedfrom an array of cameras adjusted to a variety of focal lengths, tooptimize focus of the images/video captured by the device to the workingdistance between instrument and handheld condensing lens used by theexaminer. In a preferred embodiment, these functions can be performedwithout the user having to resort to manual adjustment to mechanicallevers, dials, or switches, to improve ergonomics, enhance operationalworkflow, and minimize or eliminate the need for an assistant to adjustthe focus manually for individual users or examination parameters inwhich the lens may be held at a shorter or longer distance than typical.Additionally, the use of such techniques may enable, in one aspect,greater functionality of the device beyond the typical usage, in whichthe user can easily switch between a variant focal length or combinationof focal lengths to enable in-focus image/video capture of structures inthe anterior segment of the eye or around the eye, with or without theneed for an additional examination lens, and then switch back to normalimaging modes to focus upon the aerial image of posterior segment ocularstructures as imaged through the handheld condensing lens.

Expanded depth-of-field imaging is an algorithmic and optical set oftechniques in which one or more electronically-controlled cameras,lenses, or image sensors with different imaging characteristics (mosttypically, cameras with varying focal lengths and f-numbers/apertures)are used in combination to algorithmically join multiple images capturedat varying focal lengths into a composite image or images capturing ahigher depth of field image with higher image quality given limited orvarying ambient light than would ordinarily be captured by using onecamera or sensor in isolation.

Region of interest-based imaging is an algorithmic and optical set oftechniques in which certain preferred areas of the image(s) or videocaptured by the image sensor(s) to set global imaging settings(including, but not limited to, focus, exposure, white balance, andimage trigger control) can be reprogrammably controlled by the user viaa text- or graphical user interface on a separate networked computingdevice, or pre-set to minimize the need for user intervention. Incertain aspects, additional image processing and computational imagerecognition techniques may be used including, but not limited to,recognition of certain ocular structures or abnormal or normal clinicalfindings, to dynamically set or change associated imaging parameterswith minimal need for user intervention.

Use of onboard communications/telemetry transmission embedded in thedevice allows for multiple options for off-device transfer of data forreviewing, filing, and displaying clinical images and video. Forexample, this includes quick image review on a mobile device,simultaneous re-display on a tethered or untethered, networked videomonitor (e.g., by Bluetooth, WiFi, radio frequency, or Internetconnection), remote review by a supervising physician or other permittedthird party, remote analytics data collection, concurrent sharing ofclinical images and video with consulting practitioners (e.g.,specialists), and seamless generation of imagery and clinical metadatawhich can be scrubbed (manually or automatically/algorithmically) ofprotected health information and patient identifiers to quickly generatelarge datasets of clinical imagery and physician practice patterns,suitable for data science applications for continuous qualityimprovement, such as machine learning, artificial intelligence andprocess automation in healthcare applications, and clinical researchpurposes. The device and integrated system can enable wired or wirelessdirect transmission to electronic medical record systems andpoint-of-care billing by an examining practitioner, and concurrentpoint-of-care sharing of clinical images/video to patient and familiesof patients for medical education and facilitating continuity of care.Onboard networked computational processing capabilities allow, in oneembodiment, the off-device automatic recognition of or manual accountlogin of authorized users of the device, automating the process oflinking examining physicians to clinical imagery data for examinedpatients, examination sessions, and location. In one instance, suchnetworked sensors would allow for automatic user recognition,authentication, and tagging of generated imagery in an examinationsession via an NFC or other wireless token carried by an authorizeduser, a securely paired smartphone, or via a QR code identification tagthat, in one embodiment, would be recognized via computer visionalgorithms, using the onboard camera or cameras and embeddedmicroprocessor. Integration of onboard sensors, networked antennas, andembedded camera or cameras along with on-device and networked softwarecan allow for simple, contactless wireless network configuration andcoordination of multiple instances of the taught device within aclinical setting. The tight integration of the taught device hardware toreprogrammable on-device software and off-device software allows for agreatly improved user experience, improves usability, and enables abetter-quality clinical examination beyond prior art hardware-onlyapproaches. The device and system allows for augmented clinicalexamination facilitating review of images/videos by the practitionerwithout unnecessary discomfort to the patient following dilatedexamination, as the current clinical practice necessary to producefundus photography on a routine basis usually requires repeat serialflash photography through the dilated pupil subsequent to or prior tothe separate DFE (dilated funduscopic examination, which is conductedseparately by the examining ophthalmologist or optometrist), asconducted using a dedicated large benchtop fundus camera by anophthalmic technician in typically a physically separate clinical areafrom the examiner's medical examination room. As outlined earlier,practitioners performing the examination must remember what they see andthen either verbally describe or draw a crude picture (either by hand,or using a rudimentary painting application) of what they observed basedon their memory. The device will decrease the time necessary for suchexaminations (which are typically both inconvenient and uncomfortablefor the patient), wait time for patients currently forced to wait forthe availability of examining physicians as well as trained techniciansto capture fundus photographs, and substantially reduce the possibilityof human error due to potential for error inherent in clinicalexamination and documentation methods reliant solely upon human memoryand physician documentation.

In one aspect, the models of BIO instruments with LIO (laser indirectophthalmoscope) functionality allow simultaneous capture and remoteredisplay of imagery, as well as feature recognition/augmentedexamination and treatment (such as guiding the ophthalmologist as to thesize and region of the fundus to be treated with laser photocoagulation,the region already treated, and tracking stability vs. progression ofpathology beyond previous regions of laser treatment. The system alsoallows for remote collaboration (real-time or delayed collaboration) forlaser surgical procedures (in one application, for redisplay of barrierlaser photocoagulation of retinal breaks to help guide the lasersurgical procedure) for clinical documentation, telemedicineconsultation, and medical education purposes.

In one embodiment, the device comprises a self-contained systemcomprising one or more cameras, a beam splitter (e.g., a partiallyreflective mirror) that is at a 45-degree angle to the incoming opticalrays and the examiner so that the image being viewed by the examiner isthe same or nearly the same as the image viewed by the camera, one ormore antennas for wirelessly transmitting data to connect the devicewith one or more external devices (e.g., a computing device), useraccessible memory, and a battery. In embodiments, the camera or camerasare located in between the eyepieces of the examination instrument andunlike in some prior art designs, is not dependent upon or limited bythe field of view from only one of the binocular eyepieces. In oneembodiment, two or more cameras may be used to enable athree-dimensional or high-dynamic range view. The battery and powermanagement system may be tethered by wire or incorporated in the deviceenclosure fully, and may be user accessible for easy replacement whilein use (“hot swapping”) for extended examination periods betweencharging. The device also comprises a microprocessor, with connectedcommunications antenna system consisting of one or more types ofwireless antennas, as well as an integrated sensor array. In one aspect,the processor elements may comprise a reprogrammable system on a chip(SOC) or system on module (SOM), with one or more custom carrier PCBs(printed circuit boards), running the on-device software taught herein,bidirectionally communicating with off-device networked elements andsoftware, and comprising the non-optical functions of the device andintegrated system described. The reprogrammability of the system incombination with bidirectional networked interface with other onboard oroff-device software elements and external computing devices representadditional substantial improvements to the prior art and enable a hostof additional functionality to the integrated system, such as hereindescribed.

Image capture may be accomplished by an external foot switch, which maybe wired or wirelessly connected to the device, such as through a pairedantenna. The image may be captured by remote image trigger either by ahardware device controller, or an off-device networked software controlpaired to the specific instrument and examination session. For example,in a clinical education setting, a teacher may capture an image while astudent is using the device to examine a retina to capture a desiredimage or images of a specific feature of interest. Also, using an imagepreprocessor, sometimes in coordination with the processor, and aminiaturized camera, in combination and connected with a second or morecomputing device(s) (e.g., a mobile phone or EMR computer), auto captureis enabled along with auto enhancement, and auto-analysis, using machinelearning and computer vision techniques. Imagery (still and/or video)may be processed either on-device, or by using post-processingtechniques on a more robust connected computing device (such as, but notlimited to, mobile smartphone, networked server, cloud-connected arrayof servers, or desktop or portable computer), so as to decrease the bulkand power requirements of the described device. This innovation can beused to, in one embodiment, automatically generate a high-quality imagelibrary of the examination and of multiple serial clinical encountersover time for rapid cross-comparison and cross-reference of the imageswhile partially or wholly automating much of the ordinarilylabor-intensive process of manual association and filing ofexamination-specific elements or metadata such as the examiningphysician, patient, and date of service.

In the optical system of the device, in addition to the beam splitter,optical glare reduction techniques may be utilized, such as in oneembodiment, the use of one or more linearly polarized plates withvariable light transmission (e.g., a linear plate polarizer), which maybe used to polarize the outgoing light from the indirect ophthalmoscopeillumination source, as well as one or more additional linearlypolarized plates located prior to the camera image plane (polarizing theincoming light beams comprising the image being captured by the embeddedminiaturized camera system), to remove glare and other stray reflectivedistractions from the image optically, thereby resulting in a higherquality clinical image. These optical glare reduction techniques enabledby the physical design of the device optical system may be used inmultiple combinations with, or without, the use of dynamic algorithmicimage processing techniques to reduce glare, which are enabled by, inone embodiment, the onboard microprocessor, or, in another embodiment,off-device software hosted remotely, such as in the cloud, or on a localnetworked computing device.

In one embodiment, auto-capture comprises automatic algorithmicdetection as to whether a retina and/or optic nerve (or other fundusfeature) is in view and in focus. If that is the case, the camera willautomatically take a picture with no user intervention. This techniquecan aid in the usability of the instrument, in which there may be adelay between the user observing a feature of interest, small movementsof the eye under examination, and capture of imagery using a manualtrigger, resulting of non-capture of the desired imagery. Thiscircumstance can be further minimized by, in one embodiment, use ofconstant automatic rapid capture of a series of images heldalgorithmically in an onboard image storage buffer in the computationalsystem, such that after a manual image trigger signal generated by theuser, the desired image or images of interest can be selectedsubsequently upon review via networked off-device software, from anautomatically generated library of session images for later storage andretrieval. Auto-enhancement means that image or video capture will beautomatically enhanced, meaning the image will automatically normalizesuch features as contrast and color, and minimize glare from reflectedillumination using dynamic algorithmic image processing techniques, andincludes, but is not limited to, digital sharpening, contrastenhancement, and removal of glare and occluding elements, improving thequality of imagery beyond what the user is able to ordinarily see withan un-augmented examination instrument, and improving the ease ofcross-comparison between individual images and between clinical imagingsessions. In one aspect, the image will be centered, a standardized cropor border will be placed around the picture, and image orientation markswill be added algorithmically via on-device or off-device software.

Auto-analysis includes, but is not limited to, automatic algorithmicdetection, flagging, and measurement of clinical features of interest,and comparison with prior detected features for evidence of clinicalstability versus change, as well as comparison with reference databasesof normal and abnormal clinical features, using a variety of techniquesincluding, but not limited to, computer vision, deep learning, andartificial intelligence. In one aspect, auto-analysis will occur usingconnected software to correlate clinical images with an external libraryor set of algorithms determining attributes such as which eye is beingexamined, or flagging optic nerve and retinal periphery, noting abnormalfeatures detected by the system, all of which may aid in clinicalexamination upon review of the image(s). Though such techniques havebeen used for analysis of ophthalmic imagery using traditional funduscameras separate from the eye physician's examination, in thisembodiment, via close integration with a standard ophthalmic examinationinstrument—in this taught embodiment, the indirectophthalmoscope—algorithmic image enhancement and machine vision would bepossible in real-time or close to real-time directly from the clinicalexamination session itself, allowing the benefits of algorithmic imageanalysis of ophthalmic imagery in clinical decision-making directly atthe point of care, thus greatly improving practice efficiency and theclinical potential of any one examination and clinical encounter. Autoanalysis may also enable redisplaying image flags oralgorithmically/computationally-generated metadata in multiple formats,such as, but not limited to, text or annotated images and video. In oneembodiment, auto analysis can display its output by electronicallytransmitting metadata and clinical imagery to a connected EMR/EHR(Electronic Medical Record/Electronic Health Record) system or aseparate connected computing device or application linked to a patient'selectronic chart. In another embodiment, redisplay of auto analysisresults can be accomplished by superimposing automatically-generatedtags and/or graphical overlays illustrating areas of concern upon thecaptured imagery. Using pre- or post-image processing, the images takenduring the examination process or generated from video capture, willautomatically join photographs capturing adjacent regions of the fundus,synthesizing a montage map of the patient's fundus automatically withminimal or no user intervention, enabling cross-comparison of imagesbetween patient examinations and between patients. Such cross-comparisoncan also be conducted, in one embodiment, by quick point-of-carecross-reference to a normative or pathologic database of fundus imagery(still or video) to enable immediate or close to immediate reference ofpatient pathology to an external image library for augmented examinationenabling substantially enhanced clinical utility of the dilated fundusexamination by the use of augmented examination technology. This willmake the examination shorter and more comfortable to the patient, whilepermitting the practitioner a longer time to subsequently study thefundus by and through the images captured by the device and integratedsystem, while allowing for an enhanced level of detail, image quality,and diagnostic decision-making aids beyond which would be ordinarilypossible in un-augmented ophthalmoscopy.

The device also incorporates security features to maintain patientconfidentiality, system integrity, and integration of the device,integrated system, and other connected devices into an existing secureinformation technology network for use in a clinical setting. Videos,images, and clinical metadata, clinical data, user account information,and any other associated telemetry data pertinent to the operation ofthe device or system(s) used in the clinical data network may beencrypted by the device itself, allowing for secure transmission of datato a trusted, approved user or group of users, and allowing for ahierarchical data trail to be generated for access and manipulation ofclinical data. Physical tokens, passcodes, or connected “trusteddevices” (in one embodiment, a trusted device is a device that has beensecurely authenticated to an authorized user) can be used, in oneinstance, in combination with onboard telemetry such as, but not limitedto, networked antennas to automatically detect the presence, absence,and use of the system by a “trusted team member” (in one embodiment, atrusted team member is a user authorized to conduct a specifiedexamination or procedure and authenticated within a particular clinicalcare provider system) and appropriately tag and file generated imageryand metadata with a hierarchical audit trail to maintain data integrity,automate the appropriate tagging and filing of generated clinicalimagery and documentation, and maintain clinical data in compliance withrelevant regulatory protocols for protected health information, as wellas for clinical research data applications.

In even other embodiments, the device and system can use one or moredigital slit lamp-based camera, either integrated into or connectedremovably to a slit lamp biomicroscope, with integrated computerprocessing unit, integrated wireless antenna, and wired or wirelessconnection to on-device or off-device peripherals, controller apparati,or display systems, and wired or wireless connection to an externaldatabase. In a preferred embodiment, the system as described would serveas a “digital cockpit” for the eye care practitioner, ophthalmictechnician, or personnel conducting or documenting the eye examination,allowing for control of the system without the user diverting his or herattention away from the details of the clinical examination, by the useof wired or wireless controllers and peripherals enabling operation andadjustment of the augmented clinical examination system, and providingthe user with in-ocular (displays projected or integrated into the slitlamp biomicroscope oculars), ambient lighting, sound, haptic, and otherinteractive feedback.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate certain aspects of some of theembodiments of the present invention, and should not be used to limit ordefine the invention. Together with the written description the drawingsserve to explain certain principles of the invention.

FIG. 1 is a schematic diagram of a depiction of one possible embodimentof the device taught herein, including the overall BIO instrument (notpictured), with the portion of the overall BIO instrument including acamera, for example, pictured.

FIG. 2 is a schematic diagram of a depiction of one possible embodimentof the device taught herein, including the overall BIO instrument (notpictured), with the portion of the overall BIO instrument including acamera, for example, pictured.

FIG. 3 is a flowchart showing possible software layers used to share andstore captured and processed images.

FIG. 4 depicts generalizable models for domain shift between trainingdata and testing data.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS OF THE INVENTION

The present invention has been described with reference to particularembodiments having various features. It will be apparent to thoseskilled in the art that various modifications and variations can be madein the practice of the present invention without departing from thescope or spirit of the invention. One skilled in the art will recognizethat these features may be used singularly or in any combination basedon the requirements and specifications of a given application or design.Embodiments comprising various features may also consist of or consistessentially of those various features. Other embodiments of theinvention will be apparent to those skilled in the art fromconsideration of the specification and practice of the invention. Thedescription of the invention provided is merely exemplary in nature and,thus, variations that do not depart from the essence of the inventionare intended to be within the scope of the invention. All referencescited in this specification are hereby incorporated by reference intheir entireties.

The hardware and system taught herein allows for the ability tosemi-automatically or automatically generate fundus photographs and alive digital redisplay of the clinician's view (though not requiring anon-screen image for accurate image capture) during the conventionalworkflow of the eye exam itself, without requiring a trained technicianor separate machine to do so. This allows for each patient examinationlane to serve a dual purpose, as a combination patient examination andfundus photography/documentation station. The use of onboard andconnected computing elements allows for the rapid automation of severalelements previously requiring a dedicated technician or assistant, andphysically separate image acquisition system (such as benchtop funduscamera), in order to acquire fundus photography/clinical documentationof the posterior segment examination of the eye in addition to theclinical examination of the eye by a qualified eye care physician. Theability to conduct a retinal exam and routinely obtain a concurrentimage capture session and also enable a simultaneous, real-time,store-and-forward, or subsequent image review process, with no or almostno additional requirement for a technician or assistant to conduct thephotography, in addition to increasing clinical practice efficiencies,should enhance the clinical review process, accuracy, and interpretationof the generated images, which will increase the chances of detectingearly, subtle disease symptoms, such as early non-proliferative diabeticretinopathy (NPDR), or optic disc changes in the evaluation of glaucoma,especially compared to current examination techniques, such as using themanual BIO examination instrument alone.

Moreover, this apparatus and system makes for the ability to share,export, send, collaboratively tag, annotate, engage in remoteconsultation with colleagues, or otherwise allow for viewing outside theophthalmoscope viewfinder the images captured during or after thepatient encounter within certain medical practices or academia,including viewing remotely (e.g., telemedicine). The hardware attachment(and software interaction/system) does not interfere or minimallyinterferes in the examination process of the patient; for example, thehardware attachment should not occlude, or minimally occlude, the viewof the examiner, nor should it significantly occlude the BIOillumination source (e.g., halogen or LED light beam). In the case of abeamsplitter as part of the apparatus, the apparent brightness to theuser can be lowered or raised, but not dramatically. In other aspects,the split is 70:30 (examiner:camera light intensity ratio), which inpractice is clinically indistinguishable or minimally distinguishablefrom the traditional examination and maintains light intensity of BIOillumination well within standard illumination thresholds. In oneaspect, unlike other prior art systems, the apparatus and system do notrequire the examiner to use a screen other than what can be seen withinthe scope to verify images while capturing such images.

Capturing images will be aided by multiple redundancies. In one aspect,a redundancy is automatically built in to the system; for example, ifthe image capture does not work as programmed, the user may still usethe BIO manually as they would normally. In another aspect, if onesystem does not work as expected, or is inappropriate for a particularsetting, an alternative method will be substituted for the same task(e.g., wireless image transfer from the device's onboard electronicstorage media may not be appropriate in certain medical institutions, inwhich case a USB or SD card manual image transfer may be used instead orin addition to wireless transmission). Similarly, in some aspects,redundancies may include certain wireless networking technologies suchas, in certain aspects, Bluetooth or one or more or variations of WiFi,as well as printing images, mobile application, web-based application,computer-based application, cloud-based application, hardware-basedapplication, etc.

Image and video capture can be triggered by multiple methods includingbut not limited to via a remote app (such as on a cell phone, tablet, orcomputer), automatically, by a handswitch or footswitch, or by usingon-device or off-device microphones and voice recognition technologyconducted on-device or off-device to capture, switch imaging modes,annotate imagery, or otherwise control various elements of the deviceand integrated augmented examination system, without interfering with,or minimally interfering with, the usual indirect ophthalmologyexamination procedures conducted by the user.

In one aspect, the hardware comprises:

-   The camera(s) (in one aspect, wired to a reprogrammable    microprocessor board);-   Optional optical beamsplitter;-   Wired or wireless foot-pedal image trigger or other means of    triggering image capture;-   Embedded microprocessor, which, in one aspect, comprises a    microcomputer and connected wireless antennas and sensor array    integrated in the device enclosure, such as a wireless (in one    embodiment, Bluetooth) connection with networked computing devices    in “listening” mode awaiting input from recognized networked    devices, for use cases such as, but not limited to, image review on    an external networked computing device, or software-based    store-and-forward image sharing; and-   A power source, such as an AC (alternating current) power adapter or    battery, which may be connected to a wired or wireless (in one    embodiment, induction coil-based) charging system or “hot-swappable”    battery array system.

Additionally, in embodiments, visible and non-visible light illuminationfrom a slit lamp biomicroscope may be captured and automaticallyanalyzed by the system using a variety of computational photographytechniques such as other optical and sensor-assisted algorithmictechniques, including focus stacking, software-selectable focus planes,expanded depth of field imaging, and region of interest (ROI)-basedfocus and exposure controls to ensure focus and exposure without or withminimal user intervention in routine clinical examination settings,image enhancement, automatic montaging to see more complete pictures of,for example, the anterior segment of the eye, and annotation offindings. Focus stacking is similar to expanded depth of field imaging;also known as focal plane merging and z-stacking or focus blending; itis a digital image processing technique which combines multiple imagestaken at different focus distances to give a resulting image with agreater depth of field (DOF) than any of the individual source images.Focus stacking can be used in any situation where individual images havea very shallow depth of field, such as encountered in the anteriorsegment examination of the eye using a slit lamp biomicroscope.

In one embodiment, image processing and computational techniques may beused to use visible and non-visible light from the slit lampbiomicroscope to conduct additional qualitative and quantitative dataand analyses for the user. For example, the slit lamp biomicroscopeillumination system may be used to generate a slit beam of light focusedupon the cornea of the patient's eye under examination. The slit beamcan serve as a parallel optical pipette. The anterior and posteriorcurvature of the cornea may be detected by the onboard one or more slitlamp based camera sensors for measurement of the relative and absoluteanterior and posterior curvature of the slit beam, and computationallyanalyzed across a horizontal or vertical sweep of the patient's corneato produce a reconstruction of the patient's anterior and posteriorcorneal curvature.

In an embodiment, one or more additional sensors or emitters could beused and either reversibly attached via the use of a slit lamp adaptersystem to the stereotactic examination light path of the examining slitlamp in relation to the ocular structures examined, to enable additionaldiagnostic imaging and testing to be conducted beyond visible light slitlamp photography. The use of beamsplitters and/or in-line and/orparaxial sensor and emitter sources could be used within an integratedslit lamp-based system to conduct a variety of additional diagnostictesting simultaneously or in succession (for example rapid succession)during the same clinical examination of the patient using the same slitlamp station in the same examination lane. For example, opticalcoherence tomography, B- and A-scan ultrasound, corneal topography,corneal tomography, corneal pachymetry, scanning laser ophthalmoscopy,fluorescein angiography, fundus autofluorescence, electroretinogram, andfundus photography reversible adapters or integrated sensors and emittersystems could be used to enhance the diagnostic capabilities of thecurrently limited conventional slit lamp examination, while enhancingclinical workflows. In an embodiment, such a system could be mademodular, enabling additional diagnostic testing modules to be attachedto the system while maintaining full use of the clinical examinationslit lamp biomicroscope by the user. The use of one or morebeamsplitters could be used to divert the imaging path to maintain acoaxial or paraxial image and emission pathway with the ocularstructures examined by the user using the slit lamp.

In one embodiment, the system would allow front-to-back imaging of theeye by automatically or interactively linking imagery captured from theslit lamp-based wireless camera (for example, by an ophthalmictechnician prior to the physician's examination of the patient) as a“preview” of the anterior segment and nonmydriatic central fundus(“posterior pole”) examination not requiring pharmacologic dilation ofthe patient's pupils, with subsequent still and/or video fundus imageryof the peripheral retina captured via an integrated or adapter-basedsystem and device capturing ocular fundus imagery from the physician'sdilated fundus examination (“DFE”) of the same patient using a binocularindirect ophthalmoscope (“BIO”). The system would thus link (and in anembodiment, register and align imagery and ophthalmic structures andpathology) anterior segment and posterior segment imagery captured usinga slit lamp-based digital ophthalmic imaging system with the posteriorsegment imagery captured using a compatible BIO-based digital ophthalmicimaging system, enabling full anterior and posterior segment digitalimaging of the entire clinical examination of an individual patient fromthe clinical examination lane itself, by the use of augmentingconventional ophthalmic examination equipment and processes with anintegrated digital, wireless, smart imaging system.

In an embodiment, the system would wirelessly file diagnostic data andmetadata into a separate one or more clinical database (such as the sameone or more clinical database used by the BIO-based imaging adaptersystem).

In an embodiment, the video and/or still image stream from the devicecould be wired or wirelessly redisplayed on one or more remote displayapparatus, such as a flat-screen display panel.

Two or more digital cameras may be deployed paraxial with the imagingpath, enabling three-dimensional still and video capture of clinicalphotography and related diagnostic testing, such as binocular digitaloptic disc photographs. In this embodiment, a three-dimensional displaymay be used to display a binocular view to the examiner or viewer, suchas (but not limited to) the use of three-dimensional flat panels,binocular augmented reality displays, or binocular virtual realitydisplays.

In an embodiment, the same peripherals and processes of display,control, and manipulation of the BIO-based imaging system could be usedto control the the slit lamp-based imaging device as part of anintegrated, multi-device, augmented clinical examination system of thepatient. The use of a physical or virtual mode switch may be used, whichwould be controlled by the user on or off the device, voice controls, orautomatic detection of the portion of the eye being examined. Forexample, the use of the slit lamp-based imaging system for detection ofthe slit lamp illumination source being turned on by the user,signifying the slit lamp examination portion of the patient's eye examand tagging and filing related imagery from the slit lamp camera ascorresponding to this examination portion; and conversely, the use ofthe BIO-based imaging system to detect the user turning on the BIOillumination source and tagging and filing related imagery from theBIO-based camera as corresponding to the dilated fundus examination ofthat patient.

In an embodiment, peripherals specific to the slit lamp-based camera andimaging system could be used to control the integrated slit lamp-basedcamera and imaging system, or control various aspects of a broader,interconnected augmented clinical examination system comprising one ormore compatible imaging and diagnostic testing devices, as well asconnected bioinformatics databases, image PACS systems, telemedicine andvideo streaming systems, and/or digital authentication systems withoutleaving the clinical examination station, similarly to a pilotcontrolling different devices and systems from a central cockpit.

Similarly, in an embodiment, wired or wireless peripherals and controlsfor the slit lamp-based wireless digital imaging system may adapt (suchas, but not limited to, removable adapter attachments to the slit lampjoystick) or be integrated into the slit lamp biomicroscope joystick.These may include (but not be limited to) the use of thumbwheels, pushbutton controllers, haptic feedback controllers, trackball typecontrollers, resistive or capacitive touch controllers, switches, orknobs. Additionally, additional integrated on-device or off-devicecontroller peripherals for non touch-based control and system actuationmay be used, such as microphones for voice activated controls of thesystem.

Regarding the camera, it may exhibit exact focus correspondence (to anin-focus image manually generated by a user of the indirectophthalmoscope using a handheld condensing lens and standard indirectophthalmoscopy examination techniques), replicable in production toparallel rays and corresponding focal length to capture reflected raysfrom the beamsplitter image. In one aspect, it allows for standardizedfocal elements of the onboard camera, e.g., “number of turns” of thecamera barrel threads, to achieve focus if using a custom camera mount.In one aspect, focus is defined by lens barrel length and should not“drift” once set; preferably, the focus ring is not accessible by theuser except in a servicing situation. A variety of hardware- orsoftware-based techniques may be used to achieve consistent focuscorrespondence between the aerial view of the retina through the user'shandheld condensing lens, the examiner's view through the BIO instrumenteyepieces, and the image captured via the onboard camera, such as ahigh-depth of field camera system, such as, but not limited to, the useof an onboard lens preferably with aperture greater than f4.0, unlikemost embedded camera systems with narrow depth of field and apertureapproximately f2.0. In another embodiment, the use of multiple optical-,sensor-, or algorithmic elements may be used to conduct compositeimagery from multiple captured images using different camera settingssuch as focal planes or exposure settings, or to dynamically captureimages in rapid succession to allow for the provision of a ring bufferor set of dynamically captured images to allow the user, using the oneor more camera and one or more additional devices such as a computer ormobile device, to display short dynamic clips of video or high-frequencyimages with which to select the desired image from a series of imagesdisplayed to the user on a separate viewing screen. Electronicallytunable lenses are used, by way of example, but not limited to,dynamically adjust focus based on examination lens characteristics suchas diopter and working distance of the lens being used, user observationdistance, and user accommodation.

In one embodiment, there is exact, or almost-exact, alignment betweenthe camera, BIO view through the eyepieces, and orientation of opticalelements relative to the optical and illumination system enclosed whollywithin the BIO instrument, which in many aspects, typically comprises asingle central triangular reflecting mirror block, reflecting imagerylaterally (and again, via a second set of two triangular mirror blocks)onward to the BIO instrument eyepieces; and a superiorly locatedillumination source reflected nearly coaxial to the optical axis of theBIO instrument via another, superior but centrally-located, angledmirror. A triangular reflecting mirror block used within a standard BIOhardware system is known to those of skill in the art.

In aspects, imaging enhancements may be enabled by a two- ormultiple-camera system, such as enhanced depth of field, level of zoom,increased sharpness, expanded dynamic range, or other optical and/orsensor-based viewing enhancements of captured imagery, such as red-freefundus photography, fundus autofluorescence (FAF), or fundus angiography(FA) by the use of a second camera and specially tuned optics (such asusing specialized image gratings or sensors to capture particularwavelengths or ranges of wavelengths of light with or without the use ofintravenously-injected photo-emissive dyes such as fluorescein andindocyanine green dye, useful in clinical examination of the fundus inconditions such as diabetic retinopathy, but traditionally requiringmuch larger, bulky, dedicated imaging systems and dedicatedtechnicians).

In preferred embodiments, the camera will be neither too large nor tooheavy to significantly impair a typical examination, so as to maintaindesirable user ergonomics during the examination session. If certainembodiments are required to be larger or heavier, the camera is designedto balance weight between front and rear sections of the camera, takinginto account places where a user will touch/manipulate or where theirhair or glasses could become entangled or otherwise affect the clinicalexamination and/or image capture. In an embodiment, the hardware taughtherein will comprise an “all in one” housing design, incorporatingcamera, beamsplitter/optics, computing/processing/transmission/sensingmeans, and/or battery (or other power means (e.g., AC adapter)). In oneembodiment, the battery may be mounted externally on the user, such asat the waist (for example on a belt), with a wired attachment to themain instrument-mounted device.

In preferred embodiments, any wires, such as the camera control wire,will not be exposed outside the BIO, although in some embodiments thecontrol wire and/or other wires will be exposed.

In aspects, the optical system may comprise a separate, air-gapped smallbeamsplitter (partially-reflective mirror) to be attached or alignedover the BIO viewport. The beamsplitter, in one aspect, may be a simple70/30 plate beamsplitter, reflecting a second ray 90° from the opticalpath. The beamsplitter design, rather than a coaxial camera, may be usedin one aspect, having near-100% correspondence between the user'sexamination and the generated images with minimal requirement for theuser to have to manipulate mechanical control levers on the apparatus,or to have to make significant adjustments to achieve user-cameraoptical correspondence before, during, or after the examination intypical use cases. The user may generate focused images (parallel rays)by, most commonly, a handheld 20-diopter or 28-diopter condensing lensas the user has been trained to do in routine indirect ophthalmoscopy,taking into account factors like image tilt, corneal distortion, and apatient's refractive error of the eye examined; when the aerial image ofthe ophthalmic fundus, as viewed through the handheld condensing lens,is in focus for the user, it should be in focus for the camera. The usermay, in one embodiment, calibrate the device focus in a separate set-upprocedure before using the instrument for the first time (in oneembodiment, aided by the use of an infrared-based distance sensor) toaccount for the examiner's own refractive error (such as presbyopia) andallow sharp focus of the device camera upon the aerial image generatedby the handheld examination lens.

In some aspects, ancillary optical systems may be used or included(e.g., condensing lenses to shrink the image, or parabolic mirrors)beyond just the camera, allowing for further miniaturization and anall-in-one design, while also exhibiting technical advancements. Imagedistortion (e.g., using a parabolic mirror) may or may not beunacceptably introduced by such a system; in instances where aconsistent type of optical distortion is produced by such additionallenses or mirrors, onboard or off-device dynamic algorithmic imageprocessing techniques may be used to correct for or reduce distortion inthe captured images. In a preferred embodiment, the optics do notocclude or distort (or only minimally occlude or distort) the view ofthe examiner, or the light from the BIO illumination source emittedthrough the headlamp portion of the BIO viewport.

In some aspects, linear polarizers may be used to polarizelight-emitting diode (LED) light and images to the camera. Softwaretechniques and applications may also be used (e.g., frame doubling, highdynamic range [HDR] image processing algorithms), along with softwareimage stabilization techniques to optimize light and image qualitycaptured by the image sensor of imagery from the eye. These techniqueswould help optimize image quality and fidelity in settings wherehigh-quality image capture is challenging, given the operational need tocapture images with a low-enough shutter speed to capture a sharp imagedespite the natural movements and saccades of the patient eye beingexamined, the relatively dark background illumination of the examinationroom, the bright illumination source and numerous sources of glareduring the examination, and relatively darkly-pigmented ocularstructures in many patients. In certain examples often found in clinicalpractice, these techniques would help capture high-quality, well-exposedimagery in circumstances in which there is a large difference betweendark and bright areas of the image captured, in patients with dark fundi(relatively darkly-pigmented retina and other posterior-segmentstructures of the eye), or in patients who are quite light-sensitive andso the illumination levels used by the BIO instrument are relativelylow. Software techniques for glare reduction include, but are notlimited to, eliminating imagery captured at maximum image brightness, orclose to a 255 value of image brightness, when using grayscale pixelvalues in software image processing. To optimize image stabilization,hardware or software techniques may be used, such as automatic imagestabilization computer software hosted on the device or off-device, oroptical techniques such as isolation of the optical system to vibrationby the use of additional elements such as, but not limited to,vibration-dampening elements surrounding the optical components such asthe camera lens, image sensor, and plate beamsplitter, pivoted supportelements such as gimbal stabilizers with or without onboard motors,adaptive optical systems, or the use of internal springs mounted betweenthe device external enclosure and optical platform.

Triggering the camera to capture an image may be accomplished by way ofa manual or automatic handswitch, footswitch, or other networkedcomputing device, which may be wired or wireless. In some embodiments,other aspects of the apparatus may be placed in and/or around thefootswitch, such as a CPU, battery, processor, and/or other components.In a preferred embodiment, no user intervention would be required beyondthe conventional techniques of indirect ophthalmoscopy; for example,auto-capture of images would occur once an in-focus retina or part(s) ofthe retina are detected by a combination of integrated device elementsincluding, but not limited to, the image processor, integratedalgorithmic image processing software, additional sensors such as, inone embodiment, a distometer, and device firmware. This may becomputationally or algorithmically implemented. In one aspect, computersoftware detects whether a retina or portion of the retina is in view(e.g., based on color and/or other characteristics) and withinprogrammed tolerance limits for distance and focus, and begins and stopscapturing images automatically, either on-the-fly as high-quality fundusimagery is detected by the system by criteria such as discussed prior,or after a pre-programmed, adjustable number of images or time interval.For example, the software may stop capturing images and alert the userwhen an adequate capturing of the retina or portion of the retina isfinished. In another example, the software, with or without userfeedback to the device, may timestamp or otherwise indicate the varioustransit phases of imagery captured during the administration ofintravenous light-emissive dyes. Other criteria for starting andstopping may be used. For example, using edge detection or other imageprocessing techniques, the software may only instruct image capture whenan in-focus image is detected. Alternatively, images may be capturedusing a remote trigger via a computer software application (“an app”)either via mobile device, web-based, or other means. By way of example,a person other than the examiner may trigger image capture on a phone,tablet, or computer, and the person may be physically located at theexaminer site or in a remote location.

Upon capturing images, image export from the device onboard memory andfile storage system may occur automatically, or images may be manuallyretrieved and exported to a separate file storage system either integralto the device, or to a removable file storage apparatus (such as, forexample, SD card or a USB device). Images may also be exportedwirelessly. An automatic, user-accessible file system hierarchy may, inone embodiment, attach metadata to the captured imagery and organize theimages with little or no direct user intervention from the indirectophthalmoscopy examination session. In one aspect, these images may beencrypted to protect the health information of the patient to meetHIPAA/HiTech Act requirements. For a “store and forward” image captureand review system, systems for image capture, retrieval, review,redisplay, annotation, comparison, enhancement, and storage may includedirectly or via a compatible software integration with, in one aspect, asoftware image management, electronic health record or electronicmedical record (EHR/EMR) system, or document management system. Imagesmay, in aspects, use wireless communication protocols such as Bluetoothto send to a device, such as a mobile device, or WiFi Directtransmission to individual team members or pre-authenticatedhierarchical groups of authorized users (respectively here also referredto as “trusted users,” or “trusted groups”).

In a preferred embodiment, wireless image/data transfer is encrypted tomaintain information security and/or transmitted to trusted users only.Encryption of encoded and transmitted data may be performed on-device oroff-device, and in a preferred embodiment, network security will bemaintained across all associated networked access points and connecteddevices. In aspects, this wireless data transfer will occur via localretransmission using accepted wireless data transmission standards (suchas, but not limited to, WiFi and Bluetooth connections) to a mobiledevice, web app, or computer, such as between teacher(s) and student(s).In a preferred embodiment, radio frequencies and transmission standardsused for wireless communication between connected devices and thetransmitting device antenna array will balance goals of low-powerconnections, ease and stability of connectivity, data transmissionspeeds, line-of-sight requirements, distance of data transmissionrequired, number of wireless connections required per device, andaccepted standards and preferred radio frequencies for medical wirelessdata transmission to minimize electromagnetic interference with othertechnologies used in the medical environment. In a preferred embodiment,one-time auto-pairing between trusted user and device will happen;otherwise, sharing/pairing is set once each time the user is changed.

Regarding the microprocessor aspect of the apparatus, such as a computerprocessing unit (“CPU”), in one embodiment, the system used by thecamera and device/BIO could be a microprocessor. Another option is touse a pre-processing chip on the camera, which are, for example, used incertain webcams, which might conduct the majority of the processingeither on a mobile phone, tablet, or computer, or in the handswitch orfootswitch. In a preferred embodiment, a system on chip (SoC) or systemon module (SOM) design platform will be used; advantages include but arenot limited to reduced size, reduced cost, and easier and enhancedmaintenance capabilities of the platform. In preferred embodiments,small, lightweight processors will be used to enable processing, and inone aspect, an all-in-one single board computer system could be used. Across-compiling toolchain may also be used. Integrated softwareapplications hosted on remote networked computing devices may also actas remote controllers of the device, such as, but not limited to, aweb-based app or an app on a mobile device or computer.

The use of onboard microprocessor, integrated image processingcomputational elements, wireless networking capabilities, and integrateddevice firmware and supported software applications together enableseveral other advantages beyond the prior art. Examples of theseadvantages include, but are not limited to, on-the-fly switching betweenone or more wirelessly-connected remote displays concurrent, orsubsequent to, the clinical examination session, without interruption ofthe image capture, device operation during indirect ophthalmoscopy, orvideo stream generated by the device and user in a particularexamination session. Additionally, integrating authentication ofclinical users with a particular device, location, and examinationsession, may enable clinical collaboration significantly beyond theprior art. In one embodiment, the use of authenticated device users,wireless image and video capture and transmission, wirelessbidirectional data transfer, onboard programmable microprocessor, remoteredisplay of imagery generated in a clinical examination session, andcompatible integrated off-device software applications can enableconcurrent or time-delayed communications between clinical colleagueseither in close or remote physical proximity via a separate clinicalsoftware application. Bidirectional wireless data transfer can enable,in one aspect, remotely-triggered image capture, remote switching ofdevice modes, or remotely enabling other device functions without theuser's need for direct intervention. Examples of these advantagesinclude, but are not limited to, the eye physician user checking imagequality of the capture session after capturing imagery from one eye ontheir connected smartphone; annotating the imagery with certain notesand flagging the image for review by a senior physician and sending aquiz question based upon the imagery to a junior colleague via thesoftware application user interface; a junior resident physiciansimultaneously viewing the examination session imagery via their ownconnected mobile device to learn the techniques of indirectophthalmoscopy and for vitreoretinal medical education purposes; one ormore remotely-located physicians observing the imaging sessionconcurrently with the examination or in a time-delayed fashion, makingclinical observations into the communications interface in compatibleInternet software applications during a Grand Rounds medical educationsconference; and a senior physician verifying the quality andcompleteness of the indirect ophthalmoscopy examination, switching imagecapture modes at a certain point from a nearby networked computingdevice in order to better capture a desired ocular feature whilesignaling the mode shift to the user using audio and haptic cues, allwithout an interruption in the medical examination process.

In one preferred embodiment, the embedded microprocessor and wirelessantenna array, along with integrated, secure remote networking softwaresuch as virtual private network (VPN) software, along with algorithmictechniques such as packet forwarding, will allow pre-authenticated,authorized (“trusted”) system technicians to troubleshoot, maintain, andupdate the device or groups of devices and integrated system remotelyand provide automatic periodic updates to enhance system stability,security, and enable automatic rollout of new software-enabled functionsand enhanced functionality of the BIO digital camera system over time.This provides additional substantial advantages to the prior art, assubstantial improvements in device functionality and reliability can bemade remotely and wirelessly over time to one or more devices ordesignated groups of devices via software or firmware updates to thereprogrammable device microprocessor, and in one aspect, device errormessages or the need for system maintenance can be wirelessly sent tocompatible networked devices or off-device networked softwareapplication interfaces to signal system administrators that maintenancefunctions are necessary for a particular device (which, vialocation-based tracking as described prior, can help the systemadministrator pinpoint a specific device location and particular doctorswho may need replacement devices). In a particularly largemulti-specialty practice or hospital-based setting, these sorts ofremote device administration features enabled by the device taught hereoffer substantial advantages beyond tedious periodic inspection of eachdevice within a practice for already-busy practice personnel andancillary staff members.

The microprocessor and integrated on-device image processing, devicefirmware, and integrated off-device networked software applications, inone aspect, will allow for auto-montage of the retina and ocular fundus.Auto-montage can be described as the automatic, or software-assisted,joining, through one or more established or novel algorithmictechniques, of adjacent ocular fundus imagery captured throughindividual overlapping images or sets of images through the relativelynarrow field of view of the handheld condensing lens used in indirectophthalmoscopy, relative to the full span of the ocular fundus as viewedby the complete examination of each eye. The fundus montage would offera full “map” of the fundus of each eye in a single high-resolutionimage; have, in one aspect, the ability to compensate for distorted orincomplete capture of features in one or more individual image captures;provide an easily-magnified image that could be stored, retrieved, andcompared to prior examination montage images; and provide a “global”view of the fundus as a whole, as opposed to the narrow “keyhole” viewof individual images viewed under high magnification through thehandheld lens. The montage would typically be examined on an off-devicedisplay by the user or third-party reviewer at the end of theexamination, or at the beginning of a follow-up examination to, forexample, compare patient's examinations for progression versus stabilityof any clinical pathology found. In one aspect, a process would “stitchtogether” more than one image of the retina or ocular fundus (hereshortened to simply “the fundus”), or of other ocular features, toprovide a more complete picture of a larger portion of the retina orocular fundus than just one picture of one portion of the retina orfundus, allowing for quick review of the entire fundus at one time,rather than scanning through a library of images of successive portionsof the fundus. Several existing technologies, essentially imagestitching algorithms, may assist with the auto-montage feature, inaddition to or beyond any novel techniques used. These existingtechnologies will be familiar to one of ordinary skill in the art.On-device or post-processing enhancements may be assisted by the use ofembedded sensors on or enclosed in the device and connected peripheralelements such as accelerometers, infrared-based distance sensors, orgyroscopes, to automatically detect or suggest image orientation,portion of the fundus examined in an image frame, etc.

A sensor array may be provided on the imaging device disclosed hereinincluding one or more of the sensors as described directly above orelsewhere in the application.

The processor or other software may also allow for auto-crop andauto-center with a black frame typical of standard fundus photographs.Auto-crop may be described as the automatic, algorithmic removal ofextraneous imagery captured by the onboard system beyond the ocularfeatures desired to be captured, which may or may not be used incombination with the algorithmic placement of a standardized black framewith or without standard orientation marks as typically seen inhigh-quality fundus photographs. Auto-center may be described as theautomatic, algorithmic recognition of the circular handheld condensinglens ring held by the examiner and centration of this ring in thecaptured fundus photograph, with or without additional image processingtechniques such as adjustment for tilt of the lens (forming an ellipse,rather than a circle in the captured photograph) to minimize distortionsin the captured photograph where applicable. The images produced bythese techniques allow for easy comparison of fundus photos generated bythe system described herein to traditional fundus photographs thatexaminers typically inspect as generated by currently existing fundusphotography technology. The onboard device image processor or otherpost-processing software may also reduce/eliminate ambient lightingartifacts (such as, by way of an example, fluorescent lighting). Theprocessing or other post-processing software may include providing ared-free image or reduced-red image to better distinguish between blood,vessels, and the ordinarily orange or reddish features of the ocularfundus. The use of a reprogrammable, wirelessly networked devicemicroprocessor and integrated software applications hosted on- oroff-device allows a variety of options to exist to correct for imagerycaptured in a flipped or reversed orientation beyond the prior art,which largely must rely on optical elements to correctly orient theimagery captured, or subsequent manual manipulation of captured imageson separate computer software requiring substantial user intervention.In a preferred embodiment, correction of image or video orientation maybe performed in real time or near real time to the clinical examinationby on video processing of the camera output, which represents a standardimprovement versus direct mirror of video output. Furthermore,ophthalmic imagery captured may be in real time or near real timere-oriented by such software- and hardware-based techniques in thetaught system to their correct anatomic orientation (which is ordinarilyreversed and flipped in the aerial image as viewed through the handheldcondensing lens used in indirect ophthalmoscopy).

A variety of data science, image processing, and computer sciencealgorithmic techniques may be employed to further enhance the diagnosticand medical record-keeping capabilities of the integrated system heretaught. Algorithmic techniques such as, but not limited to, machinelearning-based automated image element recognition for the system mayalso be included as part of the device and system. Such technology maybe used to, for example, recognize that a focused retina is in view (toinitiate capture), to recognize which eye is examined, and when a higherlevel of image magnification is used or needed (for example, to capturea high-quality image of the optic nerve head of each eye), to locatelarge library/libraries of tagged fundus images (e.g., right or left)for algorithmic computer vision-based fundus photography image gradingusing computing applications and algorithms such as, but not limited to,TensorFlow, and/or rapidly collect large datasets of clinical imageryalone or in combination with clinical metadata for artificialintelligence-based healthcare automation software systems such as, toname one example, DeepMind Health. In one embodiment, optimized softwarelibraries for machine learning can be integrated with the devicemicroprocessor or image coprocessor to enable rapid acquisition ofalgorithmically enhanced or recognized imagery, balancing the power anddata storage limitations of the portable embedded system taught herein.Bidirectional data transfer and device control capabilities of theembedded system taught herein can also, in one embodiment, userecognized user and patient states based upon algorithmically-recognizedocular features to enable or simplify automatic or manual switchingbetween disparate imaging modes optimized for high-quality capture ofvarious desired structures imaged in and around the eye of the patientunder examination.

The battery and integrated device power management system, in apreferred embodiment, will feature a distinction between low-power andhigh-power states (also here referred to “sleep/wake” states), to switchfrom sleep to wake states when the user needs to use the apparatus. Forexample, in one embodiment, the device will “wake” whenever a user picksup the BIO instrument from the wall by use of onboard sensors such as anaccelerometer, thereby not requiring manually powering on the device bythe use of a physical switch each time a user puts a BIO on his/herhead. In some embodiments, the device may be charged by an AC adapter.In some embodiments the device will include a battery to support fullyuntethered use of the device. The battery may be an integratedrechargeable battery, or also could, in one embodiment, support a “hotswappable” array of user-replaceable batteries to extend use time overan extended clinical session; in one aspect, the battery may berecharged by a universal serial bus (USB) charging cable and compatiblealternating current (AC) adapter or separate power storage system. Inanother aspect, the device will be charged when placed into a dedicatedcharging station or holding base. In another aspect, the device may becharged from the BIO charging cradle either directly, by the use of acoupled charging adapter custom-fit to each model of charging cradle,such as those commonly used to charge “wireless” BIO instrument models(commonly described as “wireless,” in that they use onboard rechargeablebatteries, versus a wired AC power connection). In another embodiment,charging of the device could be conducted using an adjacent wired orwireless charging station (using, in one embodiment, wireless batterycharging technologies such as wireless induction coil-based charging),enabling the quick use and replacement of the device throughout a busyclinical workflow and the simple continued recharging of the device inbetween patient examination sessions.

Data formats for device capture, manipulation, storage, retrieval, andtransmission of data that is created and stored by the device arereferred to as, in some aspects, documents. A document may containmultiple blocks of data received from the hardware device. These blocksof data are referred to as pages. A document must contain at least 1page, but has no upper limit on the number of pages. An exception tothis is if there are errors on the device. In that case, a document withno pages can be returned, but the error collection will be filled in.Also, if there are errors, the document may still contain pages.However, these pages should be assumed to represent invalid data.Documents are grouped together in a session, which generally representsa patient exam. Sessions may contain documents obtained from multipledifferent hardware devices. Each session, document, and page within thedocuments may have searchable metadata that is not patient identifiable.This is to provide a quick means of searching the data captured andstored without having to decrypt every record.

In one aspect, the basic structure may appear as follows. A session maycomprise, but is not limited to: globally unique ID; MRN, which inpreferred embodiments is either an encrypted value, or a hash of a valuethat is stored elsewhere; Name; Description; Start Timestamp; EndTimestamp; Responsible User IDs; Documents; and Device Groups associatedwith the Session.

A Document may comprise, but is not limited to: globally unique ID;Device ID; Operator ID; Pages; Metadata entries; and Messages. Documentswill contain at least one Page or Message.

A Page may comprise, but is not limited to: globally unique ID; DataType Description; Data Format; Filename; DataURI (Data Uniform ResourceIdentifier), in a preferred embodiment the raw data is stored in adifferent data source to keep actual patient data isolated fromidentifying data; Timestamp; Metadata entries.

A Metadata entry may comprise, but is not limited to: globally uniqueID; Key; and Value.

A Message may comprise, but is not limited to: globally unique ID;Device ID; Device Message ID; Message Severity; Message Type; Text; andTimestamp.

A Device Group may comprise, but is not limited to: globally unique ID;Name; Description; Session ID; and Devices.

A Device may comprise, but is not limited to: globally unique ID; Name;Description; DeviceType; and Device Group ID.

A DeviceType may comprise, but is not limited to: ID and Name.

In other embodiments, different file formats may encode other types ofmetadata. Different devices or device versions may encode, in certainembodiments, the state of the software configuration, the state of thedevice, and calibration data, that excludes patient-specific orprotected health information. In certain embodiments, these may includean IP address of the device, the software and hardware version of thedevice, the device geolocation of the device, or other device- oruser-specific data useful to device management and interpretation of thedata but not specific to the patient-related data itself.

In embodiments, the above data format may be used for:

Sessions

Sessions, in embodiments, may be analogous to patient visits. There willtypically be a one-to-one correspondence between a session and a patientvisit. In aspects, sessions are not connection instances since a webservice may be used; accordingly, connections may not necessarily beheld open. Sessions can be created, started, stopped, etc. via awebsite, for example. This allows the workflow of associating devicedocuments with a session that is not associated with an MRN, and thenallowing a user to manually split that data into sessions linked toMRNs. For example, a user could use the timestamps of the documents.

Documents

Documents may comprise, in aspects, a single “upload” of data from aDevice to a Session. In aspects, a Document will be one Page; forexample, a fundus image. In other aspects, a Document will comprisemultiple pages, allowing for, for example, a burst of images from thedevice and uploading the images in one step, saving different fileformats, saving other types of data along with the data (e.g., audio,left and right eye images, etc.). In one embodiment, a Document may besent from a device to a Session that contains only a Message. Messages,in one aspect, comprise a data type well-suited for devicestatus-related information such as, for example, “Battery Level,” asthey include, in aspects, Acuity and Message Type. In this example, ifthe device onboard battery is at 20% charge, the user may receive aWarning Message, but at 5% charge, the user could, in this embodiment,receive a Critical Message. In this embodiment, as the device detects alow battery charge, the system may send a device status-related messagewithout being restricted by whether the user captures an image.

Pages

Pages, in aspects, comprise raw data from the Device, along with dataformat info for storage information.

Metadata

Metadata, in aspects, is used to store “out of band” data that may beuseful to the user. This may include, but is not limited to, calibrationdata, software/hardware versions, geolocation data, etc.

Devices

Device records, in aspects, may comprise data for the user and theserver to identify a device that is registered with the system.

Device Groups

Device Groups, in aspects, comprise collections of Devices that can beassociated with a Session. For example, this may be used to set up allDevices in a particular examination room to be grouped in a DeviceGroup. When a user creates a Session for a patient, the information maykey, for example: “Patient will be in Examination Room B; associateExamination Room B Device Group with the Session.” Accordingly, alldevices in Examination Room B are automatically linked to the Sessionand will send accompanying Documents to the correct data location.Device Groups, in aspects, may contain a single device if it is desiredthat devices not be grouped together.

The system taught herein may also be used as part of or in electronic(wired or wireless) bidirectional connection with a paired datamanagement control device (here referred to as a “hub”) to manage andorganize devices, examinations, and people involved in the examinationprocess and associated data produced by each correctly and across theclinical data network. The hub comprises a processor (e.g., a CPU) andmay be connected to the Internet with wire(s) or wirelessly. It may alsobe unconnected from the Internet in a dedicated local area network (LAN)or wide area network (WAN). In a preferred embodiment, the hub will bewirelessly connected to devices or examiners in the examining facilityto monitor activity and permit multiple device control and coordination.In one aspect, the hub will receive images and data/information from thedevice taught herein or other devices. It will be used, along withuniquely identifiable markers such as hardware tokens, paired mobiledevices, or passcodes, to detect and manage the hierarchy of trustedusers engaged in use of a connected network of devices as previouslydescribed. It will break down the data, review the data, analyze thedata, manage for storage, sync images and information, process images orinformation, and/or manage remote data synchronization and/or local orremote redisplay. It may also manage storing such information locally orremotely.

For example, the hub may be connected to several devices taught hereinwithin a facility. The hub will record when such devices are being usedand who is using the devices to automatically, or with minimal userintervention, maintain a secure hierarchical audit trail of clinicaldata generated by the clinical data network here described. The hub willlog, save, organize, and process such information in order to, amongother things, know when examinations were being performed, what kind ofexaminations were being performed, and who was performing suchexaminations. Personnel in or around the facility may be tracked, in oneaspect, by having a radio-frequency identification (RFID)-or near fieldcommunications (NFC)-compatible tracking device on their person, or bytracking a paired mobile phone or some other electronic devicewirelessly by a variety of hardware-assisted and algorithmic softwaremethods such as, but not limited to, location-based and time offlight-based wireless tracking. The information collected by the hub maybe cross-referenced with other information, for example a referenceclinical schedule, to track activity in or around the facility. Inanother example, the hub may automatically, or with user input, pairpatient imagery and metadata collected during an examination with thespecific patient and associated doctor at that appointment time, withthe hub subsequently associating and exporting collected data to thepatient's EMR/EHR based on the reference clinical schedule used.

Components in or on the BIO include, but are not limited to, a batterycharger, an accelerometer, a wireless or wired charging means orconnection, a WiFi or Bluetooth chip (wireless network chip), a battery,a microprocessor, an antenna(s), a microphone, a speaker, a powercontroller, switches (for example, for on/off or other commands), andstatus LED(s) (for example, for indicating on/off, error, ready forcertain commands, etc.). The status LEDs will be designed, in apreferred embodiment, to be viewed at a distance, while remainingunobtrusive during the eye examination in a darkened examination room,while the BIO instrument is positioned in, for example, a wall-mountedbattery charging cradle.

Now turning to the Figures, FIG. 1 shows a computer-generated image ofthe BIO portion with camera and electronic elements (the rest of the BIOinstrument is not shown for clarity, as one of ordinary skill in the artis familiar with general aspects of a BIO instrument separate from theinvention disclosed and claimed herein). In one aspect, one or morecamera is situated 1400 central and above the main elements of the BIO.The camera view, in aspects, is centrally coaxial and identical with theexaminer's view. In embodiments on the top of the aspect of the BIOinstrument, as pictured, an area exists for additional components,including a processor, antenna, storage medium, etc., althoughcomponents may be located elsewhere on the BIO or not on the BIO. Suchcomponents include, but are not limited to, a battery charger, anaccelerometer, a wireless or wired charging means or connection, a WiFior Bluetooth chip (wireless network chip), a battery, a processor, anantenna(s), a microphone, a speaker, a power controller, switches (forexample, for on/off or other commands), and status LED(s) (for example,for indicating on/off, error, ready for certain commands, etc.). Thestatus LEDs will be designed, in a preferred embodiment, to be viewed ata distance, while remaining unobtrusive during the eye examination in adarkened examination room, while the BIO instrument is positioned in itswall-mounted battery charging cradle.

FIG. 2 shows a front-facing view of a BIO portion including the imagingaspect according to embodiments herein, portions of the BIO instrumentitself 6100, camera 6400, processor 6900, and antenna/battery/sensorarray configuration 6901 (the rest of the BIO instrument is not shownfor clarity, as one of ordinary skill in the art is familiar withgeneral aspects of a BIO instrument separate from the inventiondisclosed and claimed herein). The processor, wireless antenna, battery,sensor array allow for image/video transmission and/or processing,system control via multiple mechanisms (mechanical, voice, mobileapplication, local or remote computer control, handheld orfootswitch-type remote controller), and bidirectional data transmissionand telemetry.

FIG. 3 shows a flowchart of the high-level architecture of the systemsoftware application(s) used to implement the intended purpose of theimaging BIO taught herein. Specifically, a block diagram is showndepicting data transmission and conceptual hierarchy or software layersand services. In one embodiment, at the command receiver level, theimage captured by the device(s) 3000 will be sent by Bluetooth, WiFi, bywire, or by OnDevice listeners. The data may be encrypted 3001. Next,the command service function is performed, wherein a command bus handlesinfo command and session command 3002. Clinical data capture sessionsare then managed and if there is a current session, information flows tothe session repository proxy 3003 and then to the data storage layer3004 and information such as documents, information, images, and/ormetadata are stored in a session repository before moving to maintenance3005. At the maintenance level, the documents, information, images,and/or metadata are backed up to cloud-based storage 3006.

Further Aspects of Machine Learning and Artificial Intelligence:

This system aims to extend glaucoma (or other pathology) screeningtechnology access to underserved populations at greatest risk with avariety of new and refined tools. First, is the use of a binocularindirect ophthalmoscope-mounted wireless digital imaging adapter whichenables diagnostic image capture and redisplay of the fundus examinationfrom the examination lane itself; second, the development of refined AImethods for automated detection of the optic nerve and featuresegmentation in digital photographs suitable for a diverse array ofcamera types of variable image fidelity; and third, the automation ofDDLS risk scoring in real-time during the eye examination at the pointof care on a local device. As the physician examiner of at-risk patientsordinarily has little ability for in-lane fundus photography without theuse of a diagnostic examination instrument-based or -mounted imagingsystem, the diagnostic examination is generally a “black box” topatients, and the status quo promotes multiple transcription andinterpretation errors between examination and clinical documentationsteps, due to the lack of clinical photography for the majority ofpatient encounters during the comprehensive eye exam. The current systemas described here produces automated, interpretable, device-agnosticdetection of eye diseases, such as but not limited to glaucoma, toovercome existing usability challenges. The related algorithms can uselow-fidelity imagery with artifacts found in common diagnosticinstruments, and the algorithmic AI/ML model may be, in an embodiment,optimized to run in a performant fashion on a low-powered mobile devicesuch as (but not limited to) a mobile smartphone, tablet, desktop orlaptop computer. In another embodiment, and AI/ML model may be optimizedto run on the device itself, and not require a stable connection to theInternet or additional networked computing devices beyond the localimaging system and paired computing device. Additionally, the use ofdigital adapter systems mounted on or integrated into existing clinicalexamination tools (such as, but not limited to, BIO-based imagingsystems or slit lamp-based imaging systems) may significantly increasethe throughput of available images collected through the eye examinationprocess by mounting or integrating with commonly-deployed and -useddiagnostic examination instruments used worldwide by eye carepractitioners and their clinical personnel. Increasing throughput ofimage capture may further increase the robustness, performance, anddatabase size of diagnostic testing and imaging databases used fortraining AI/ML models. Finally, by integrating with digitaldocumentation and authentication technologies by the user with clinicalexamination tool-based augmented examination systems, such a system mayalso automate or streamline secure data and metadata tagging andinterpretation steps currently requiring human intervention andjudgment, which is a common bottleneck in training AI/ML models for usein healthcare settings.

The currently described system also may enable automation of physicianlabeling/validation using text natural language processing (NLP) and/orpoint of care (POC) voice recognition; diagnostic multi-modal imagingprospectively for multiple disease types and causal inference(CI)/integration research regarding captured pathology; and enhanceddiagnostic accuracy at the point of care using existing diagnosticinstruments; and clinical workflow and human computing interface (HCI)studies.

In an embodiment, the system could utilize a database of synthetic lowquality clinical images (with synthetically generated imaging artifactssuch as encountered in the clinical examination); existing high-qualitycolor, red-free, or false-color digital fundus images with labeledmetadata may be used, along with related raw data of clinical encounternotes (such as ICD-10 codes/clinical notes).

Generalizable AI/ML models may be used to account for domain shiftsbetween training data and testing data. In a related embodiment,domain-generalizable models would be used across imagingdevices/population shifts in clinical data used to iteratively train andtest the generalizable models, to ensure fair prediction regardless ofpatients' race/skin color and access to devices.

In a related embodiment, AI/ML algorithmic models for ophthalmicstructure, feature, and segmentation may be used, and the modelsre-trained using the synthetically generated low-fidelity digitalimages, to generate new generalizable models. The generalizable modelsand related algorithms would then be used to process a library ofophthalmic images of patients using slit lamp-based or indirectophthalmoscope-based digital adapters, and labeling the artifact andpathology regions either algorithmically or by the user and the resultspaired and entered into an associated bioinformatics database.Additionally, in an embodiment, the algorithms and models could bemodified to screen out low quality images, identify artifacts, as wellas the region of true abnormality in ophthalmic imagery captured by theuser. In an embodiment, the system could additionally sort capturedimagery into multiple levels of quality (such as, in an embodiment,sorting, tagging, and displaying captured imagery as sufficient orinsufficient quality for analysis). In another embodiment, the model andrelated algorithms would classify and sort large bioinformaticsdatabases of clinical images (such as, for example, true color or falsecolor fundus images) to confirm a normal distribution of fundus typesand pigmentation levels.

In an additional embodiment, the system could allow for user interactionand intervention at key steps, such as but not limited to confirmation,adjustment, or rejection of automatically detected, segmented, andanalyzed ophthalmic structures. This would further increase user andpatient confidence, clinical utility, and understanding with the AI/MLsystem used by making key clinical steps available and interpretable tothe user.

In an additional embodiment, the system of generalizable models andalgorithms could export generated qualitative and quantitative data toand from separate clinical decision support (CDS) computer softwaretools, image registration and montage software, image PACS systems, andbioinformatics databases. This would enable multi-modal imaging ofophthalmic imaging, as well as qualitative and quantitative analysis ofphysiologic and pathologic features over time—for example, to identifydisease or risk level progression over time for a patient by theintegrated analysis of a variety of data types and sources. (See, e.g.,FIG. 4.)

One skilled in the art will recognize that the disclosed features may beused singularly, in any combination, or omitted based on therequirements and specifications of a given application or design. Whenan embodiment refers to “comprising” certain features, it is to beunderstood that the embodiments can alternatively “consist of” or“consist essentially of” any one or more of the features. Otherembodiments of the invention will be apparent to those skilled in theart from consideration of the specification and practice of theinvention.

It is noted in particular that where a range of values is provided inthis specification, each value between the upper and lower limits ofthat range is also specifically disclosed. The upper and lower limits ofthese smaller ranges may independently be included or excluded in therange as well. The singular forms “a,” “an,” and “the” include pluralreferents unless the context clearly dictates otherwise. It is intendedthat the specification and examples be considered as exemplary in natureand that variations that do not depart from the essence of the inventionfall within the scope of the invention. Further, all of the referencescited in this disclosure are each individually incorporated by referenceherein in their entireties and as such are intended to provide anefficient way of supplementing the enabling disclosure of this inventionas well as provide background detailing the level of ordinary skill inthe art.

1. A binocular indirect ophthalmoscope system, comprising: a binocularindirect ophthalmoscope comprising one or more camera and at least oneof a computer processing unit, a memory unit, or a communication device;and one or more remote electronic device; wherein the at least one ofthe computer processing unit, the memory unit, or the communicationdevice is capable of sending (a) one or more electronic still imageand/or video image and (b) information associated with the one or moreelectronic still image and/or video image, from the binocular indirectophthalmoscope to the one or more remote electronic device by way of awired or wireless connection; wherein at least one of the computerprocessing unit or the one or more remote electronic device, allow foraltering at least one of settings on the one or more camera, operationof the one or more camera, or control of the one or more camera, and/orwherein the at least one of the computer processing unit or the one ormore remote electronic device, allow for triggering capture of the oneor more electronic still image and/or video image; and wherein the atleast one of the computer processing unit or the one or more remoteelectronic device, are further capable of providing for or allowing foroptical-and/or sensor-assisted algorithmic techniques chosen from one ormore of the following: a. focus stacking; b. removing or reducing glareor visual occluding elements from the electronic still image and/orvideo image; c. automatic capturing of the one or more electronic stillimage and/or video image when at least one of a retina, an optic nerve,or another portion of an eye is detected and in focus; d. comparing theone or more electronic still image and/or video image captured by theone or more camera with a library of images to detect one or more ofwhich eye is being examined, an optic nerve, a retinal vasculature, amacula, or a retinal periphery; e. comparing the one or more electronicstill image and/or video image captured by the one or more camera with alibrary of images to detect one or more abnormal feature of an eye beingexamined; f. software-selectable focusing planes; g. expanded depth offield imaging; h. region of interest-based focusing; i. dynamic orautomatic exposure controlling to ensure proper exposure without or withminimal user intervention in routine clinical examination settings; j.dynamic or automatic focus controlling to ensure proper focus without orwith minimal user intervention in routine clinical examination settings;k. construction of one or more three-dimensional view of an eye orocular structure; l. construction of one or more high dynamic rangeimage of an eye or ocular structure; m. automatic registration of theone or more still image and/or video image captured of a similar portionof a fundus over one or more imaging session; n. automatic montaging ofoverlapping adjacent portions of the one or more still image and/orvideo image for a more complete representation of one or more of aretina, a fundus, or another portion of an eye; o. electronic annotationof observations; or p. machine learning to automatically detect one ormore ophthalmic structure, anatomic features, or pathology, for at leastone of automated segmentation, registration, or image qualityenhancement, to calculate qualitative and/or quantitative diagnosticmeasurements for use of clinical decision support tools to aid in one ormore of diagnosis, screening, or treatment.
 2. The binocular indirectophthalmoscope system of claim 1, further comprising a sensor array. 3.The binocular indirect ophthalmoscope system of claim 1, furthercomprising a footswitch controller to trigger capturing the one or moreelectronic still image and/or video image using the one or more camera.4. The binocular indirect ophthalmoscope system of claim 1, wherein theremote electronic device is chosen from a phone, a computer, a tabletcomputer, a server, a laptop computer, a television, a monitor, asensor, a second computer processing unit, Internet, local areanetwork-, or wide area network-connected device.
 5. The binocularindirect ophthalmoscope system of claim 1, wherein the at least one ofthe computer processing unit or the remote electronic device, are usedto perform one or more of the following tasks: a. control settings on orrelated to the one or more camera; b. to operate the one or more camera;c. to redisplay the one or more electronic still image and/or videoimage captured from the one or more camera; or d. to transmit, receive,analyze, gather, or collect information associated with the one or morecamera, the one or more electronic still image and/or video imagecaptured from the one or more camera, or both.
 6. The binocular indirectophthalmoscope system of claim 1, wherein the one or more still imageand/or video image are processed by at least one of locally storedcomputer software applications, computer software applications storedremotely on a separate networked device, or computer softwareapplications stored on a remote server.
 7. The binocular indirectophthalmoscope system of claim 1, wherein the one or more cameraprovides an extended focal plane, field of view, or both, and is capableof capturing the one or more electronic still image and/or video imagecorresponding to portions of a human or animal eye as viewed by a userof the binocular indirect ophthalmoscope.
 8. The binocular indirectophthalmoscope system of claim 1, wherein the one or more camera ispositioned above and between the eyepieces of the binocular indirectophthalmoscope, and wherein the one or more camera is mounted centrallyor paracentrally to incident light beams in a visual axis of thebinocular indirect ophthalmoscope.
 9. The binocular indirectophthalmoscope system of claim 1, further comprising a lens orcombination of lenses having an f-number greater than f-1.8.
 10. Thebinocular indirect ophthalmoscope system of claim 1, further comprisingcomponents communicating one or more of visual or nonvisual ambientnotifications to a user of the binocular indirect ophthalmoscope chosenfrom one or more of visual cues, audio cues, or haptic feedback.
 11. Thebinocular indirect ophthalmoscope system of claim 1, wherein thebinocular indirect ophthalmoscope is capable of enabling one or more ofbi-directional networked image or video capture, redisplay, control, ortransmission of the one or more electronic still image and/or videoimage.
 12. The binocular indirect ophthalmoscope system of claim 1,wherein the binocular indirect ophthalmoscope is capable of enablingbi-directional networked automation of one or more of: filing ororganization of the one or more electronic still image and/or videoimage; or association of the one or more electronic still image and/orvideo image with a specific patient, an examining physician, a date ofan examination, a specific ocular feature detected by the binocularindirect ophthalmoscope, a specific binocular indirect ophthalmoscopeused, image quality data, operational data of the binocular indirectophthalmoscope, or a location of the binocular indirect ophthalmoscopeor examination.
 13. The binocular indirect ophthalmoscope system ofclaim 1, wherein adjustment knobs or levers on the binocular indirectophthalmoscope allow for adjusting one or more of: a. interpupillarydistance; b. transmission or manipulation of an illumination source ofthe binocular indirect ophthalmoscope; c. instrument illuminationintensity, aperture, and/or angle of the binocular indirectophthalmoscope; d. optical alignment; e. stereopsis; f. viewing angle;or g. position of the binocular indirect ophthalmoscope on a user'shead.
 14. The binocular indirect ophthalmoscope system of claim 1,wherein the at least one computer processing unit is capable ofintegrating information associated with one or more of an eyeexamination using the binocular indirect ophthalmoscope, a patient's eyebeing examined, a patient being examined, or an examiner examining thepatient's eye and the one or more electronic still image and/or videoimage to perform one or more of the following: diagnose the patient;treat the patient; schedule another examination or appointment;communicate with the patient, the examiner, or third parties; or createa database or library of all or part of integrated informationassociated with the eye examination, the patient's eye being examined,the patient being examined, the examiner examining the patient's eye andthe one or more electronic still image and/or video image, orcombinations thereof.
 15. The binocular indirect ophthalmoscope systemof claim 1, further comprising a multiple-camera system, wherein themultiple-camera system is coaxial or paracentral with a viewing axis ofa user providing for binocularity of recorded imagery, and wherein themultiple-camera system is capable of enhancing recorded imagery.
 16. Thebinocular indirect ophthalmoscope system of claim 1, wherein enhanceddepth of field, level of zoom, increased sharpness, expanded dynamicrange, or other optical and/or sensor-based viewing enhancements ofcaptured imagery, are enabled by using the one or more camera, one ormore image sensors, or optics to capture wavelengths of light from afundus with or without the use of photo-emissive dyes, and with orwithout the use of an illumination source.
 17. The binocular indirectophthalmoscope system of claim 1, wherein sensor- or user-assistedfeedback timestamps indicate transit phases of the electronic stillimage and/or video image imagery captured during an administration ofphoto-emissive dyes.
 18. The binocular indirect ophthalmoscope system ofclaim 1, wherein the electronic still image and/or video image arecaptured with or without user feedback, including indication of patienteye laterality and examination orientation of a portion of a fundusbeing examined, which assist in automatic or partially-automated joiningof multiple fundus images captured using the one or more camera.
 19. Thebinocular indirect ophthalmoscope system of claim 1, whereincomputational photography algorithmic techniques digitally combinemultiple images into a composite image or capture images set at variouscombinations of aperture and focal length settings.
 20. The binocularindirect ophthalmoscope system of claim 1, further comprising one ormore of distometers, time of flight sensors, or additional rangecameras, which provide feedback data to the one or more camera,including compensation factors for one or more of user working distance,lens type, observation distance, or level of accommodation oraccommodation lens add, for use in setting focus or other camerasettings.
 21. The binocular indirect ophthalmoscope system of claim 1,further comprising one or more of visible and invisible light spectrumimage sensors, distometers, mechanical sensors, or antennas, in an arraycapable of providing feedback to assist with one or more of automatedfocusing, field of view, recognition of a patient, recognition of auser, recognition of ocular imagery, or triangulation of device locationand orientation.
 22. The binocular indirect ophthalmoscope system ofclaim 1, further comprising an adaptive optics system, wherein theadaptive optics system is capable of compensating for distortions oraberrations in an optical viewing pathway.
 23. The binocular indirectophthalmoscope system of claim 1, further comprising electronicallytunable lenses, wherein the electronically tunable lenses are capable ofadjusting focus based on examination lens characteristics, including oneor more of diopter and working distance of a lens being used, userobservation distance, or user accommodation.
 24. The binocular indirectophthalmoscope system of claim 1, wherein a calibration image, includinga printed calibration reference card, a digital image, or set of imagesviewed on a separate screen, is used to set or reset device settings,including one or more of focus settings, alignment of a camera imagewith a user's point of view through the binocular indirectophthalmoscope, or adjustment of the binocular indirect ophthalmoscopeto confirm a same image is being viewed through both eyepieces of thebinocular indirect ophthalmoscope.
 25. The binocular indirectophthalmoscope system of claim 1, further comprising embedded oroff-device microphones, wherein the embedded or off-device microphonescan be used, in combination with computational and algorithmic elementson the binocular indirect ophthalmoscope or off the binocular indirectophthalmoscope, to detect and recognize one or more of voice commands,one- or two-way communication with remotely located telemedicine users,or annotation of imagery.
 26. The binocular indirect ophthalmoscopesystem of claim 1, further comprising an illumination source within anon-visible spectrum of light, wherein the illumination source is usedto flash light to register focus or capture imagery through anonmydriatic pupil.
 27. The binocular indirect ophthalmoscope system ofclaim 1, wherein multiple images are captured in rapid successionallowing for a ring buffer of still or video imagery, or allowing for aset or sets of a plurality of captured images to allow a user of thesystem to display clips of video or high-frequency images for selectinga desired image from a series of images displayed to the user.